Evaluating Various Tokenizers for Arabic Text Classification

The first step in any NLP pipeline is to split the text into individual tokens. most obvious and straightforward approach use words as However, given a large corpus, representing all not efficient terms of vocabulary size. In literature, many tokenization algorithms have emerged tackle this problem by creating subwords, which turn limits size corpus. Most techniques are language-agnostic, i.e., they do incorporate linguistic features language. Not mention difficulty evaluating such practice. paper, we introduce three new for Arabic compare them other popular tokenizers using unsupervised evaluations. addition, six on supervised classification tasks: sentiment analysis, news poem-meter classification, publicly available datasets. Our experiments show that none best choice overall performance algorithm depends factors including dataset, nature task, morphology richness dataset. some better compared others various tasks.